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0^ 1 Abstract. This is a brief survey of recent results by the authors devoted to 

one of the most important operators of integral geometry. Basic facts about 
the analytic family of cosine transforms on the unit sphere in R n and the corre- 
■^1^ sponding Funk transform are extended to the "higher-rank" case for functions 

■ on Stiefel and Grassmann manifolds. The main topics are the analytic continu- 

ation and the structure of polar sets, the connection with the Fourier transform 
on the space of rectangular matrices, inversion formulas and spectral analy- 
, sis, and the group-theoretic realization as an intertwining operator between 

generalized principal series representations of SL(n,R). 



1. Introduction 

<N 

04 ■ The cosine transform has a long and rich history, with connections to several 

branches of mathematics. The name cosine transform was adopted by Lutwak 
50, p. 385] for the spherical convolution which is defined on the unit sphere S™ _1 
0> in R n by 

(Cf)(u)= f f(v)\u-v\dv, uer 1 . (1.1) 

The motivation for this name is that the inner product u ■ v is nothing but the 
cosine of the angle between the unit vectors u and v. 
. The following list of references shows some branches of mathematics, where the 

$_i ' operator (jl.ip and its generalizations arise in a natural way (sometimes implicitly, 

without naming) and play an important role. 

• Convex geometry: [Tl [6l [23l [24l [32l [46l [50l [69l [TTl [75] . 

• Pseudo-Differential Operators: [151 161) . 

• Group representations: [2l [3l [TTl fT2l [571 [60] . 

• Harmonic Analysis and Singular Integrals: [4[ [^[^[^l48[[5^[[58l[59[[63[ 
166117311741179]. 

• Integral geometry: |5l l20l [26l l30l l62l l64l l65l l68l 1701 ITS! [86] . 

• Stochastic Geometry and Probability: [29l |49"1 UTTl [77l [78] . 
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• Banach Space Theory: [38" l IIS] \47[ l54j 172] . 

This list is far from being complete. In most of the publications cosine-like 
transforms serve as a tool for certain specific problems. At the same time, there are 
many papers devoted to the cosine transforms themselves. The present article is 
just of this kind. Our aim is to give a short overview of our recent work [571 170) on 
the cosine transform and explain some of the ideas and tools behind those results. 

For a complex number A, the A-analogue of the operator (jl.lj) is the convolution 
operator 

(C x f)(u)= [ f(v)\u-v\ x dv, ue S n_1 , (1.2) 
JS"- 1 

where the integral is understood in the sense of analytic continuation, if necessary. 
We adopt the name "the cosine transform" for (|1.2p too. The same name will be 
used for generalizations of these operators to be defined below. 

In recent years more general, higher-rank cosine transforms attracted consid- 
erable attention. This class of operators was inspired by Matheron's injectivity 
conjecture |51) . its disproval by Goodey and Howard [29 , applications in group 
representations [7J 1121 1571 1601 186] and in algebraic integral geometry [3], [5], 120) . 
To the best of our knowledge, the higher-rank cosine transform was explicitly pre- 
sented (without naming) for the first time in [261 formula (3.5)]. 

As mentioned above, the present article gives a brief survey of recent results 
by the authors [571 170) in this area. The consideration grew up from specific 
problems of harmonic analysis and group representations. However, we do not 
focus on those problems, and mention them only for better explanation of the 
corresponding properties of the cosine transforms and related operators of integral 
geometry. Here we shall restrict ourselves to the case of real numbers, referring to 
the above articles for the case of complex and quaternionic fields. 

The paper is organized as follows. Section 2 contains basic facts about the 
cosine transforms on the unit sphere. More general higher-rank transforms on 
Stiefel or Grassmann manifolds are considered in Section 3, where the main tool 
is the classical Fourier analysis. In Sections 4 and 5 we discuss the connections to 
representation theory, and more precisely to the spherical representations and the 
intertwining properties. Section 6 is devoted to explicit spectral formulas for the 
cosine transforms. 



2. Cosine transforms on the unit sphere 

In this section we discuss briefly the cosine transform on the sphere S n_1 . We keep 
the notation from the Introduction. For the analytic continuation of the cosine 
transform it is convenient to normalize it by setting 



(tf A /)(«) = 7n (A) / f(v)\u-v\ x dv, u£S n -\ 

Here dv stands for the SO(n)-invariant probability measure on S Tl_1 and the nor- 
malizing coefficient 7 n (A) is given by 

^ nnmmiTmy ReX> -^ ^o,m,.... (2 .i) 
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This normalization is chosen so that 



r( - A/2) 



r((n+A)/2) 

Such a normalization is convenient in many occurrences, when harmonic analysis on 
the sphere is performed in the multiplier language (in the same manner as analysis 
of pseudo-differential operators is performed in the language of their symbols). We 
shall see below that it also simplifies the spectrum of the cosine transform. 

The limit case A = — 1 gives, up to a constant, the well-known Funk transform. 
Specifically, if / € C^ 1 ), then for every u € S n ~\ 

*i^^ = 1^572) ™ (tt) - (2 ' 2) 

where 

(Ff)(u)= J f(v)d u v. (2.3) 

{uGS"- 1 |n-u=0} 

In (I2.3p . d u v stands for the rotational invariant probability measure on the (n — 2)- 
dimensional sphere u ■ v = 0; see, e.g., [691 Lemma 3.1]. 

The operators C x and c £ x were investigated by different approaches. A first one 
employs the Fourier transform technique [451 1631 176] and relies on the equality in 
the sense of distributions 

ExC x f r \ / E-x-nf 



ci = 2 n+X TT {n - 1 ^ 2 T(n/2). 
Here w is a test function belonging to the Schwartz space S(M. n ), 

and (E\f)(x) = \x\ x f(x/\x\) denotes the extension by homogeneity. 

A second approach is based on the Funk-Hecke formula, so that for each spher- 
ical harmonic Y) of degree j, 

'^y, »'„aV,. (2.5) 

where 

f (_i),7a r(j72 -A/2) 
mj, x =< 1 ' r(j/2 + (n + A)/2) J (2.6) 
[ if j is odd; 

see, e.g., |63j . The Fourier-Laplace multiplier {mj t \} forms the spectrum of e tf x . 
Note that the normalizing coefficient in < ^ pA was chosen so that only factors depend- 
ing on j are involved in the spectral functions {TOj.a}. The spectrum of ( t? x encodes 
important information about this operator. For instance, since mj t \nij t —\— n = 1, 
then for any / 6 C°°(S™ -1 ) the following inversion formula holds: 

tf- x - n tf x f = f, (2.7) 

provided 

A G C, A £ {-n, -n-2, -n-4, ...}U {0,2,4, . . .}. 



4 



G. OLAFSSON, A. PASQUALE, AND B. RUBIN 



For the non-normalized transforms, (|2.7|) yields 

C^f = C (A) /, C (A) = T - {n/2) m + A)/2) r((1 - A - n)/2) 



(2.8) 



7iT(-A/2)r((n + A)/2) 

AeC, \i {-l,-3,-5,...}U{l-n,3-n,5-n,...}. 

Formula ()2.6j) reveals singularities, provides information about the kernel and 
the image. Moreover, it plays a crucial role in the study of cosine transforms on 
V functions. For instance, the following statement was proved in |641 p. 11], using 
the relevant results of Gadzhiev 12 1|, 122] and Kryuchkov |48j for symbols of the 
Calderon-Zygmund singular integrals operators. 

Theorem 2.1. Let L^S™ -1 ) and L^ e (S n_1 ) be the spaces of even functions (or 
distributions), belonging to L p (S n ) and the Sobolev space L^(S™ -1 ), respectively. 
Then 



(2.9) 



provided 
7 = Re A - 



(n-1), 



S = Re A 



(n-1), 



\£ {0,2,4,...}U{-n-l,-n-3, -n 
The embeddings \2. 9\) are sharp. 



5,...}. 



Finally, one can use tools from the representation theory, as we will discuss in 
more details in the second half of this article. 

One can easily explain (|2.5[) - but not (|2.6p - by the fact that the space of 
harmonic polynomials of degree j is the underlying space of an irreducible rep- 
resentation of K = SO(n). Then (|2.5| follows from Schur's lemma and the fact 
that c £ x commutes with rotations. Note that the group K acts by the left regular 
representation on L 2 {S n ~ 1 ) and, as a representation of K, we have the orthogonal 
decomposition 

i'tO-if®^. (2-10) 

where the set 3^ J of all spherical harmonics of degree j is an irreducible K-spa.ce. 
As we shall see in Section El the spectral multiplier (|2.6j) can also be computed by 
identifying ^ A as a standard intertwining operator between certain principal series 
representations of the larger group SL(n,R), see [57] . 

The fact that c £ x is zero on the odd power harmonics follows from the obser- 



vation that the kernel 



is an even function of v. Hence ^ is actually an 



integral transform on the projective space P(K ra ), and here the analogue of ()2.10j) 
is 

L 2 (p(R n )) —k y j ■ 

jG2N 



3. Cosine transforms on Stiefel and Grassmann manifolds 

In this section we introduce the higher-rank cosine transforms and collect some 
basic facts about these transforms. The main results are presented in Theorems 
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3.1. Notation. We denote by V„ im ~ 0(n)/0(n — m) the Stiefel manifold 
of n x m real matrices, the columns of which are mutually orthogonal unit n- 
vectors. For v £ V„. m , dv stands for the invariant probability measure on V„. m ; 
£ = {v} denotes the linear subspace of 1" spanned by v. These subspaces form the 
Grassmann manifold G n>m ~ 0(n)/(0(n — m) x O(m)) endowed with the invariant 
probability measure d£. We write M„ jt „ ~ R ram for the space of real matrices 
x — (xij) having n rows and m columns and set 

n rn 

dx — Y\ dxij, \x\ m = Aet{x t x) 1 ^ 2 , 
i=i j=i 

x* being the transpose of x. If n — m, then |x| m is just the absolute value of the 
determinant of x\ if m = 1, then x|i is the usual Euclidean norm of x s R". 

3.2. The Cos-function. We give two equivalent "higher-rank" substitutes for 
\u-v\ in The first one is "more geometric" , while the second is "more analytic" . 
For 1 < m < k < n — 1, let rye G n>m and £ G G n ,k be linear subspaces of R™ of 
dimension m and fc, respectively. Following [2J [3], I57j . we set 

Cos(e,?7) =vol m (Pr c S), (3.1) 

where vol m (-) denotes the m-dimensional volume function, E is a convex subset of 
•q of volume one containing the origin, Pr^ denotes the orthogonal projection onto 
£. By affine invariance, this definition is independent of the choice of E. 

The second definition |31l gives precise meaning to the projection operator 
Pr^. Let u and v be arbitrary orthonormal bases of £ and 77, respectively. We 
regard u and v as elements of the corresponding Stiefel manifolds Y n ,k an d V„. m . 
If k = m = 1, then u and v are unit vectors, as in (|1.1[> . The orthogonal projection 
Pr^ is given by the k x k matrix uu l , and we can define 

Cos(£, 77) = Cos({w}, {v}) = (detiv'uu'v)) 1 / 2 = \u*v\ m . (3.2) 

This definition is independent of the choice of bases in £ and r\ and yields \u ■ v\ if 
k = m = 1. 

Remark 3.1. Note that v t uu t v is a positive semi-definite matrix, and therefore, 
det(i?*ww*?j) = dci(u t vv t u) > 0. It means that Cos(£,7?) = Cos(?7,£) > 0. 

3.3. Non-normalized cosine transforms. According to (13. 1[) and (|3.2I) . one 

can use both Stiefel and Grassmannian language in the definition of the higher-rank 
cosine transform, namely, 

(C^ k f)(u)= f f^lu'vtdv, u€V n , k , (3.3) 

(Cm,fc/3(0=/ /WCos A (£,ry)^, £eG„, fc , (3.4) 

where dv and dr7 stand for the relevant invariant probability measures. The fact 
that we have two ways of writing of the same operator, extends the arsenal of 
techniques (some of them will be exhibited below). Both operators agree with C x 
in (|1.2p . when k — m — 1. For brevity, we shall write C^, = C^ m . 

We remark that there are different shifts in the power A in the literature, all for 
different reasons. In particular, to make our statements in Sections 2-4 consistent 
with those in |70j . one should set A = a — k. To adapt to the notation in [57] one 
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has to change A to \ — n/2. For unifying the presentation of the results in [70j and 
[57] we have preferred to adopt the unshifted notation as in (|3.3p and (|3.4[) . 

Following |16l 128) , the Siegel gamma function of the cone fi of positive definite 
to x to real symmetric matrices is defined by 

m— 1 

r m (a)= / exp(-tr(r))M«-( m+1 >/ 2 dr = n™^/ 4 J[ T(a-j/2) (3.5) 
Jn 1=0 
and represents a meromorphic function with polar set 

{(m-l-j)/2 | j = 0,1,2,...}. (3.6) 

Theorem 3.2. Let l<m<k<n — l. 

(i) ///Gi^Vn^) and Re A > to — k — 1, then the integral \3. 3\) converges 
for almost all u£V n ,k- 

(ii) If f £ C°°(V njm ), then for every u G V n>k , the function A H> {C* n k f){u) 
extends to the domain Re A < to — k — 1 as a meromorphic function with 
the only poles to — k — l,m — k — 2, ... . These poles and their orders are 
the same as of the gamma function r m ((A + k)/2). 

(iii) The normalized integral (C^ k f)(u)/T m ((X + k)/2) is an entire function 
of X and belongs to C°°(V n ^k) in the u-variable. 

A similar statement holds for (I3.4j) . The proof of Theorem 13.21 can be found 
in |70i Theorems 4.3, 7.1]. It relies on the fact that is a special case of the 

composite power function (u*w) A with the vector- valued exponent A G C m |16L 128] . 
The corresponding composite cosine transforms were studied in |58|, 1591 170) . 

An important ingredient of the proof of Theorem l3.2l is the connection between 
the cosine transform k f on V„. m and the Fourier transform 

0{y) = (JV)(2/) = / e^ ytx ^(x) dx, y e M n . m . (3.7) 

-'M„, m 

The corresponding Parseval equality has the form 



(<p,w) = (2n) nm (<p,uj), fau)= l (p{x)u(x)dx. (3.8) 

This equality with w in the Schwartz class S(M„ tm ) of smooth rapidly decreasing 
functions is used to define the Fourier transform of the corresponding distributions. 

We will need polar coordinates on M„ jrn , so that for n > to, every matrix 
x e M„ m of rank to can be uniquely represented as x = vr 1 / 2 with v £ V„ jm and 

r = x t x G fl. Given a function / on V n , m , we denote {E\f)(x) = \r\m /(«)• The 
following statement holds in the case k = m. 

Theorem 3.3. Let f be an integrable right O(to) -invariant function on V„. m , 
uj G S(M n ^ m ), 1 <m<n— 1, C^f = C„ m f. Then for every A G C, 

E\Cmf t \ ( E-\-nf 



r m ( { x + m )/2y^) =c {r^FA72-y")> (3J) 

_ 2 m (™ +A )7r Iim / 2 r m (n/2) 
r TO (m/2) 

where both sides are understood in the sense of analytic continuation. 
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The formula (|3.9[) agrees with ()2.4[) . The more general statement for arbitrary 
fc > to can be found in [70] . 

Remark 3.4. It is important to note that the domains, where the left-hand 
side and the right-hand side of of (13.91) exist as absolutely convergent integrals, 
have no points in common, when m > 1. This is the principal distinction from 
the case to = 1, when there is a common strip of convergence — 1 < Re A < 0. 
To perform analytic continuation, we have to switch from to the more general 
composite cosine transform with A G C m and then take the restriction to the 
diagonal Ai = • • • = A m = A + to. This method of analytic continuation was first 
used by Khekalo (for another class of operators) in his papers [391 1411 140) on Riesz 
potentials on the space of rectangular matrices. 

3.4. The Funk transform. The higher-rank version of the classical Funk 
transform (|2.3j) sends a function / on V n ,m to a function F mt kf on V n ,k by the 
formula 

{F m , k f){u)= [ f(v)d u v, ueV n , k . (3.10) 

The condition u v — means that subspaces {u} £ G Uj k and {v} € G„, m are 
mutually orthogonal. Hence, necessarily, k + m < n. The case k = to, when both / 
and its Funk transform live on the same manifold, is of particular importance and 
coincides with (|2.3|) when k = m= 1. We denote F m — F m<m . 

If / is right 0(m)-invariant, (F m ^f)(u) can be identified with a function on 
the Grassmannians G n . m or G n .„_ m , and can be written "in the Grassmannian 
language". For instance, setting £ = {v} £ G„, m , r\ — {u} 1 - e G„ )n _fc, and /(£) = 
f(v), we obtain 

(R m ,n~kf)(v)= J = (F m ,kf)(u). (3.11) 

3.5. Normalized cosine transforms. Our next aim is to introduce a natural 
generalization k f of the normalized transform p.ip . "Natural" means that we 
expect ^"^j j./ to obey the relevant higher-rank modifications of the properties 



Definition 3.5. Let 1 <m <k <n — 1. For ueF^ and ueT4, m , we define 

(^m,*/)(«) = 7n,m,fc(A) / /(«) |«*«|^ d«, (3.12) 

where 

r m (m/2) r m (-A/2) 
7n,m,fc(A) = — ?7V~rTwoT ' A + to 7= 1, 2, . . . . 

r m (n/2) r m ((A + fe)/2) 

We denote = c € m m . The integral p,12|) is absolutely convergent if Re A > 
to — fc — 1. The excluded values of A belong to the polar set of T m (— A/2). If 
fc = to = 1 this definition coincides with (|2.1|) . Operators of this kind implicitly 
arose in |26l pp. 367, 368]. 



Theorem 3.6. Let 1 < to < fc < n — 1, fc + to < n. If f is a C°° right 
O(to) -invariant function on V n . m , then for every uSV n> fc, 

o.c « fc /) (u) = T ,f m/ *l , (F m ,kf)(u), (3.13) 
A=-fe r m ((n-fc)/2) 
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where "a.c." denotes analytic continuation and (F m kf)(u) is the Funk transform 

This statement follows from |70l Theorems 7.1 (iv) and 6.1]. Note that if 
m = k = 1, then (|3.13[) yields (|2.2I) . However, unlike (|2.2[) , the proof of which 
is straightforward, (|3.13[) requires a certain indirect procedure, which invokes the 
Fourier transform on the space of matrices and the relevant analogue of (|3.9[) . 

We point out that a pointwise inversion of the Funk transform can be obtained 
by means of the dual cosine transform, which is defined by 



{C A m ,kV)(v)= ^(«)|«'«|* du, veV n , m . (3.14) 
Jv n , h 

Indeed, the following result holds. 

Theorem 3.7. (cf. [7D1 Theorems 7.4]) Let (p = F m>k f, where f is aC°° right 
O(m) -invariant function on Y n ,m, 1 <m < k <n — m. Then, for every v € Y n ,m, 

ac (Cj^)(v) _ T m (n/2) 
x= m -nT m ((X+k)/2) nh r m (fc/2)r m (m/2)' { ' ' 

Regarding other inversion methods of the higher-rank Funk transform (which 
is also known as the Radon transform for a pair of Grassmannnians) , see [311 185] 
and references therein. 

In the case k = m the normalized cosine transform m = m has a number 
of important features. If / € C°°(V„ )m ), then analytic continuation of (^m/)( M ) 
is well-defined for all complex A ^ {1 — m, 2 — m, . . .} and belongs to C°°(V n m ). 
The following inversion formulas hold. 

Theorem 3.8. (cf. [701 Theorems 7.7]) Let f e C°°(V„, m ) be a right 0{m)- 
invariant function on Y n ,m, 2m < n. Then, for every u G V„. m , 

(tf- x - n Vif)(u) = f(u), \,-\-n${l-m,2-m,...}. (3.16) 

Ln particular, for the non-normalized transforms, 

(C m x - n C^f)(u) = ((X)f(u), \ + n,-\ <£ {1,2,3,...}, (3.17) 

where 

r(x] _ r^(n/2) r m ((m + A)/2) T m ((m - A- n)/2) 
U J r^(m/2)r m (-A/2)r m ((n + A)/2) " [ ' > 

Both equalities \3.1b)) and h3.18\) are understood in the sense of analytic continua- 
tion. 



In the case m = 1, the formulas (|3.16l) and (|3.17[) coincide with (|2 . 7[) and ([2 

respectively, but the method of the proof is different. 

4. Connection to Representation Theory 

The cosine transform is closely related to the representation theory of semisimple 
Lie groups. In particular, as we shall now discuss, it has an important group- 
theoretic interpretation as a standard intertwining operator between generalized 
principal series representations of SL(n, M). 

In the following we shall use the notation G — SL(n,R), K — SO(n), and 

b) B€0(»-m)' det(A)det(i?) = l| 



L = S(0(m)xO(n-m))= 
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with to < n — to. Then B = K/L = G n ,m is the Grassmanian of 771-dimensional 
linear subspaces of K™. We fix the base point 

b a = {(xi,...,x m ,0, ...,0) I xx,...,x m e M} € B, 

so that B — K ■ bo and every function on £> can be regarded as a right L-invariant 
function on K . 

From now on, our main concern is the nonnormalized cosine transform Q3.3P 
with equal lower indices, that is, C^ n = C* l m . We refer to |35l Chapter V] for 
harmonic analysis on compact symmetric spaces and [42] for the representation 
theory of semisimple Lie groups. 

4.1. Analysis on B with respect to K. The first connection to representa- 
tion theory is related to the left regular action of the group K on L 2 (B) by 

(£(k)f)(b) = f(k- 1 b), keK,beB. 

For an irreducible unitary representation (n, V^) of K, we consider the subspace 

V? := {v e K I Tr(k)v = v Vfc e L}, L = S(0(m) x 0(n - to)). 

The representation (it, V v ) is said to be L-spherical if V£ 7^ {0}. As B = K/L is a 
symmetric space, the following result is a consequence of |35i Chapter IV, Lemma 
3.6]. 

Proposition 4.1. If (ir, V*) is L-spherical, then dim V£ = 1. 

Since ^ {0}, we can choose a unit vector S V^P. Then we define a map 
<£ n : V* -> C°°(/3) C i 2 (/3) by the formula 

($*«)(&) :=d(7r)- 1 / 2 (7j,7r(fc)e 7r ), dgV,, b = k ■ b a e B = K ■ b Q , (4.1) 

where d(7r) = dimV^-. This definition is meaningful because k ■ b = kkl ■ b for 
every k' £ L and remains fixed under the action of 7r(fc'). We also set 

:= (**«)(&). 

Recall, if (77, V^) and (a, 14) are two representations of a Hausdorff topological group 
H , then an intertwining operator between 7r and cr is a bounded linear operator 
T : V n -» 14 such that T7r(ft) = cr(ft)T for all h E H. If tt is irreducible and T 
intertwines 7r with itself, then Schur's Lemma states that T — cid for some complex 
number c, [171 . p. 71. The map is a if-intertwining operator in the sense that 
it intertwines the representation n on V„ and the left regular representation I on 
L 2 (B), so that for b = h ■ b Q and k £ K we have 

^(ttOJtj; 6) = (vr(fc)7;,7r(/7)e 7r ) = (v, 7r(fc- 1 /i)e 7r ) = ^(fc)$ 7r (7j; 6) . 

Furthermore, the left regular representation £ on L 2 (B) is multiplicity free, see 
e.g. |84[ Corollary 9.8.2]. Therefore, since (%, 14) is irreducible, any intertwining 
operator V % — > L 2 (B) is by Schur's Lemma of the form 0$^ for some c £ C. 

We let L 2 (B) — Im*^. Denote by Kl the set of all equivalence classes of 
irreducible L-sphcrical representations (7r,V^) of K. Then, see [351 Chapter V, 
Thm. 4.3], the decomposition of L 2 (B) as a K- representation is as follows. 

Theorem 4.2. L 2 (B)~ K L%{B) . 
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The cosine transform is, as mentioned before, a K- intertwining operator, i.e., 
C*M k )f) = e ( k )C™(f) for all k G K and / € L 2 (B). It follows by Schur's Lemma 
that for each 7r G Kl there exists a function rj^ on C such that 

C*| L 2 = % (A)id. (4.2) 

Let / e L%(B) of norm one. Then rj n (A) = (C A (/),/) and it follows that r/^A) is 
nieromorphic; cf. Theorem 13.21 



4.2. Generalized spherical principal series representations of G. The 

fact that is a if -intertwining operator does not indicate how to determine the 
functions In the case to = 1 and in some particular cases for the higher-rank 
cosine transforms [581 159) explicit expression for r/^ can be obtained using the 
Funk-Hecke Theorem or the Fourier transform technique. It is a challenging open 
problem to proceed the same way in the most general case, using, e.g., the relevant 
results of Gelbart, Strichartz, and Ton- That, see, e.g., |25L I80|, I82j . Below we 
suggest an alternative way and proceed as follows. 

To find rj T explicitly, we observe that the cosine transform is an intertwining 
operator between certain generalized principal series representations (tt\, L 2 {B)) of 
G = SL(n, K) induced from a maximal parabolic subgroup of G. We can then use 
the bigger group G, or better its Lie algebra, to move between if -types. We invoke 
the spectrum generating technique introduced in [Tj to build up a recursion relation 
between the spectral functions % . This finally allows us to determine all of them 
by knowing Jftrivial- 

The group G = SL(n, M) acts on B by 

9 ■ V ■= {g v I v G i]} , 

where gv denotes the usual matrix multiplication. This action is transitive, as the 
if-action is already transitive. The stabilizer of b a is the group 



P 



A X 
B 



^^-BeGi-il) and d et(^)det(B) = 1 

~ S(GL(m) x GL(n - to)) x M m , n _ ro , 

where M n _ m is the space of n x to real matrices; see Section 13.11 We then have 
B = Gjp'. 

The if- invariant probability measure on B is not G- invariant. But there exists 
a function j : G X B — > M + such that for all / G L l (B) we have 

/ f(b)db= [ f(g-b)j(g,b) n db, g e G, beB. (4.3) 

JB JB 

We include the power n to adapt our notation to [57] . By the associativity of the 
action we have j(gg', b) = j(g, g' ■ b)j(g', b) for all g £ G and b € B. Hence, for each 
A G C we can define a continuous representation ir\ of G on I? (B) by 

[7TAC9)/](&):=j(fl- 1 ,6) A+n/a /(ff- 1 -6) ) / G L 2 (B), (3 G B. (4.4) 

A simple change of variables shows that 

(^(g)f 1 h) L 2^(f,TT_ J (g- 1 )h) L 2, g eG, f,heL 2 (B). 

In particular, tt\ is unitary if and only if A is purely imaginary. The representations 
n\ are the so-called generalized (spherical) principal series representations (induced 
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from the maximal parabolic subgroup P), in the compact picture. See e.g. [42] . p. 
169. 

The representations ir\ can also be realized on Stiefel manifolds as follows. 
According to [701 Section 7.4.3], we introduce the radial and angular components 
of a matrix x G M„ iin of rank m by 

rad(z) = (xfx) 1 ! 2 G 0, ang(x) = x(x t x)~ 1 ^ 2 G V„ )TO , 

so that x — ang(x) rad(x). Given A G C, we define a mapping which assigns to 
every 5 G GL(n,R) an operator Tt\(g) acting on measurable functions / on V n ,m 
by the rule 

tta(<?)/(«) = IradO/-^)!-^/ 2 ) /(ang^-^)). (4.5) 
Clearly, n\(I n ) is an identity operator. One can prove that if / is a measurable 
right 0(m)-invariant function on V n , m , then 

^(5192)/ = 7TA (51)^(52)/, 5i,32 S GL(ra,R). (4.6) 

The restriction of 7ta to SL(n, M), acting on the space of square integrable right 
0(m)-invariant functions on V n , m , coincides with the representation defined by 

63). 

4.3. The cosine transform as an intertwining operator. In this section 
we follow the ideas in [57] . An alternative self-contained exposition (without using 
the representation theory of semisimple Lie groups) , can be found in [70j . 

The gain by using the representations tt\ is that we now have a meromorphic 
family of representations on L 2 [B) and that they are irreducible for almost all A 
and closely related to the cosine transform. For this, we recall some results from 

Theorem 4.3 ( Vogan- Wallach) . There exists a countable collection {p n } of 
non-zero holomorphic polynomials on C such that if p n {\) ^ for all n then tt\ is 
irreducible. In particular, tt\ is irreducible for almost all X E C 

PROOF. This is Lemma 5.3 in [83]. □ 

Let : G — > G be the involutive automorphism 8(g) — (g" 1 )'. We remark that 
in [57] notation Cos A = Cm n ^ 2 was used. 

Theorem 4.4. The cosine transform intertwines tt\ and 7r_> o 6, namely, 

C m O 7T A+n/2 - (7T_ A _„ /2 O 9) o C ml (4.7) 
whenever both sides of this equality are analytic functions of A. 

PROOF. We refer to Theorem 2.3 and (4.10) in [57]. □ 

In fact, it is shown in [57] . Lemma 2.5 and Theorem 4.2, that C m = J (A), 
where J(A) is a standard intertwining operator, studied in detail among others by 
Knapp and Stein in [431 144] and Vogan and Wallach in [83] , These authors show, 
in particular, that A J(A) has a meromorphic extension to all of C. Furthermore, 
Vogan and Wallach show that if / G C°°(B), then the map 

{A G C I Re (A) > -1 + n/2} 9 A 1 — ► J(A)/ G C°°{B) 

is holomorphic. As a consequence of C„ n/2 = J(A) and 83, 1.6 Thm], we get the 
following theorem. 
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Theorem 4.5. The map A i— > C m extends meromorphically to C. In particular, 
for f 6 C°°(B) the function A i— > (/)(&) extends to a meromorphic function on 
C cmd i/ie sei of possibles poles can be chosen independent of f. In the complement 
of the singular set we have C m (f) S C°°(B). 

Notice that precise information about analiticity of more general cosine trans- 
forms, including the structure of polar sets, is presented in Theorem 13.21 above. 

The implication of (|4.7I) is that C m A ™^ 2 o C m ™^ 2 intertwines tt\ with itself 
(in the sense of meromorphic family of operators). By Theorem 14.31 there exists a 
meromorphic function 77 on C such that 

C m A -"/ 2 o C m - n/2 = ry(A) idc-(B) (4.8) 

for all A 6 C for which the left-hand side is well defined. The shift by n/2 in the 
definition is chosen so that the final formulas agree with those in |57j and make 
some formulas more symmetric. The fact that r\ is meromorphic follows by noting 

that v (x) = (c m x - n,2 ~o c m - n/2 (i), 1). 

Formula (|4.8[) is a symmetric version of (|3.18j) with A replaced by A — n/2. 
The explicit value of 77(A) can be easily obtained from (|3.17j) . An alternative, 
representation-theoretic method to compute the function 77(A), is presented in Sec- 
tion [6l The first step is the following lemma. 

Lemma 4.6. Let c(A) = c4~ ,l/2 (l). Then 77(A) = c(A)c(-A). 
Note that c(A) is nothing but r) trivia i(X) in Theorem 16.61 

Remark 4.7. There are several ways to prove the meromorphic extension of 
the standard intertwining operators. The proof in 183 j uses tensoring with finite 
dimensional representations of G to deduce a relationship between C m and C m +2n . 
In fact, there exists a family of (non-invariant) differential operators D\ on B and 
a polynomial b(X), the Bernstein polynomial, such that 

b(X)C m (f)=C m +2n (D x (f)) (4.9) 

1831 Thm. 1.4]. Another way to derive an equation of the form (|4.9I) is to convert the 
integral defining C m into an integral over the orbit of certain nilpotent group N, as 
usually done in the study of standard intertwining operators, and then use the ideas 
from [8j I55L 156] . In the case where G/P is a symmetric i?-space (which contains 
the case of Grassmann manifolds), the standard intertwining operators J(A) have 
been recently studied by Clerc in [9], using Loos' theory of positive Jordan triple 
systems. In particular, Clerc explicitly computes the Bernstein polynomials 6(A) 
in (|4.9p . and, hence, proves the meromorphic extension of J(A) for this class of 
symmetric spaces. 

Finally, one can stick with the domain where A H> C m is holomorphic and 
determine the JsT-spectrum functions 77^ (A) in (|4.2[) . As rational functions of T- 
factors, these functions have meromorphic extension to C. Hence, A 1— > C m itself 
has meromorphic extension by (|4.2|) . We will comment more on that in Remark 
EH 

4.4. Historical remarks. We conclude this section with a few historical re- 
marks. The standard intertwining operators J(A), as a meromorphic family of 
singular integral operators on K or N, have been central objects in the study of 
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representation theory of semimisimple Lie groups since the fundamental works of 
Knapp and Stein |43L I44j . Harish-Chandra |33j . and several others. In our case 



Then, in the realization of the generalized principal series representations on L 2 (B), 
the kernel of J(A) is Cos A ~"/ 2 (6, c). But in most cases there is neither an explicit 
formula nor geometric interpretation of the kernel defining J(A). 

Apart of customary applications of the cosine transform in convex geometry, 
probability, and the Banach space theory, similar integrals turned up independently 
as standard intertwining operators between generalized principal series representa- 
tions of SL(n, K), where K = R, C or H. 

The real case was studied in [12] , the complex case in |14j , and the quaternionic 
case in [60j . In these articles it was shown that integrals of the form 



with some modification for K = C or i, define intertwining operators between 
generalized principal series representations induced from a maximal parabolic sub- 
group in SL(n + 1,K). The if-spectrum was determined, yielding the cases of 
irreducibility and, more generally, the composition series of those representations. 
Among the applications, there were some embeddings of the complementary series 
and the study of the so-called canonical representations on some Riemannian sym- 
metric spaces of the noncompact type, [101 II XL 113) . However the connections of 
these considerations to convex geometry, to the cosine transform and to the Funk 
and Radon transforms was neither discussed nor mentioned. These connections 
were first published in [57 in the context of the Grassmannians over R , C and 
H. However, it was probably S. Alesker who was the first to remark in the unpub- 
lished manuscript [2] that the cosine transform is a SL(n, R)-intertwining operator. 
It was also shown in [8 6) that the Sin A -transform (a transform related to the sine 
transform) can be viewed as a Knapp-Stein intertwining operator. This was used to 
construct complementary series representations for GL(2n, R). The Sin A - transform 
is then also naturally linked to reflection positivity, which relates complementary se- 
ries representations of GL(2n,R) to the highest weight representations of SU(n, n), 
QS [III [53], [33 [36] ■ Notice, however, that the definition of the Sin A -transform in 
[86] differs from the one in [66], [70j : see also |67j for the sine transform on the 
hyperbolic space. 



The functions t)„(\) in (|4.2j) are parametrized by the L-spherical representations 
of K . The main purpose of this section is to present this parametrization, which 
is given by a semilattice in a finite dimensional Euclidean space associated with a 
maximal flat submanifold of B. We will, therefore, have to study the structure of 
the symmetric space B. We refer to |81) and the books by Helgason [341 135] for 
more detailed discussions and proofs. To bring the discussion closer to standard 
references in Lie theory we also introduce some Lie theoretical notation which we 
have avoided so far. 
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Let 

3 = {Xe M n , n I ix(X) = 0} , 
E = {le M„,„ | X* = -X} , 

be the Lie algebras of G — SL(n,R) and K = SO(n), respectively. The derived 
involution of 8 on g, still denoted 8, is given by 8(X) = —X 1 . Hence, t = q(1,6), 
the eigenspace of 8 on g with eigenvalue 1. We fix once and for all the G- invariant 
bilinear form /3(X, Y) = m ( n "L m ) tr(XY) on g. Note that (3 is negative definite on J 
and {X, Y) = —f3(X, 8{Y)) is an inner product on g such that ad(X)* = — ad(0(X)), 
where, as usual, ad(X)Y = [X, Y] = — YX. The normalization of /3 is chosen 
so that it agrees with |57j . 

We recall that B is a symmetric space corresponding to the involution 

r(x) = (^ ° W Im ° ) = ( A ~ B ) for U B 



-l n - m ) V -In-my/ V~ C D / V C ^ 

where for r G N we denote by I r the r x r identity matrix. Note that r in fact 
defines an involution of G and that the derived involution on the Lie algebra q is 
given by the same form. 

We have 6 = [ © q where [ ~ so(m) x so(n — m) is the Lie algebra of L and 



q = «(-!, r) = Q(X) 



0mm 

- Y t n 

y\ \Jn-—m.n—m 



X G M„ 



Let = (Si U Sj^)ij denote the matrix in M mi „_ m with all entries equal to but 
the [y 1 /i)-th which is equal to 1. For t = {t\, . . . , i TO )' G R m we set 

m 

X(t) = — ^""^ tjE n -2m+j,j € M m-n _ m , 

i=i 

y(t) = Q(x(t))eq. 

Then b = {F(t) | t G M m } ~ K m is a maximal abelian subspace of q. 

To describe the set Kl we note first that B is not simply connected. So we 
cannot use the Cartan-Helgason theorem [351 p. 535] directly, but only a slight 
modification is needed. Define ej(Y(t)) := itj. We will identify the element A = 
Sjli -\j e j W1 th the corresponding vector A = (Ai, . . . , A m ). 

If H G b, then ad(£f ) is skew-symmetric on t with respect to the inner product 
( • , • ). Hence ad(-ff) is diagonalizable over C with purely imaginary eigenvalues. 
For a G ib* let 

Eg : = {Xetcl (V-ff G b) ad(if)X = a(if)X} 

be the joint a-eigenspace. Let 

A e = {a G ib* | a ± and 6g ^ {0}} . 

The dimension of tg is called the multiplicity of a (in tc)- 

Lemma 5.1. FFe ftave 

A{ = {±£i ± (1 < i ^ j < m , ± independently), ie, (1 < i < m) } 

with multiplicities respectively 1 ( and not there if m = 1), 2n — m ( and not there if 
m = n — m). 
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Proof. This follows from [34] : the table on page 518, the description of the 
simple root systems on page 462 ff. and the Satake diagrams on pages 532-533. □ 

We let 

A^ = {ei ± ej (1 < i < j < m), a {1 < i < m)} . 

Lemma 5.2. Let p t = - ^ dim({g)a G «b*. Then pt = (— - ijej. 

«eA+ j=i 

Let now (n, V„) be a unitary irreducible representation of K. Then V v is 
finite dimensional. Moreover, tt(H) — — iv(exp(tH)) is skew-symmetric, hence, 



dt 



t=o 



diagonalizable, for all H 6 b (in fact, tt(H) is diagonalizable for all H G I). Let 
r(7r) C ib* be the finite set of joint eigenvalues of n(H) with H € b. For p 6 T(ir), 
let C K denote the joint eigenspace of eigenvalue ^i. If X £ tg and v £ VP, 
then tt(X)v G V^ I+a . Thus, there exists a p — p n G T(n) such that n(tg)V? = {0} 
for all a £ A»*. This only uses that tt is finite dimensional, but the irreducibility 
implies that this p is unique. It is called the highest weight of n. Finally we have 
7r ~ a if and only if p w = p a . 

Let K be the universal covering group of K. Then r lifts to an involution f on 
K, L := LT r is connected, and B := K/L is the universal covering of B. Replacing 
K by K etc., we can talk about L-spherical representations of K and their highest 
weights. The following theorem is a consequence of the Cartan-Helgason theorem 
p. 535]. 



(Va G A+) eZ+|. (5.1) 



Theorem 5.3. The map 7r h- > /i^ seis itp a bijection between the set of L- 
spherical representations of K and the semi-lattice 

(a, a) 



A+(B) = G ib* 
Furthermore, if m = n/2, then 



A + {B) = {(mi, . . . , p m ) G Z m | Ml > M2 > ■ • • > Mto _i > |/im|} • 

A+(B) = {(jh, . . .,p m ) eZ m \pi>p 2 >---> Hm-i > Mm > 0} . 

If p G A + (£>), then we write (ir^, V^) for the corresponding L-spherical repre- 
sentation. Recall the notation $ ff from (|4.ip . Let A + (£>) denote the sublattice in 

A + (i3) which corresponds to L-spherical representations of K. Then G A + (£>) 
if and only if the functions (v), which are originally defined on B, factor to 
functions on B. For that, let v G VP and H G b. We can normalize w and e,r M so 
that 



e 



The same argument as for the sphere [81[ Ch. III. 12], proves the following theorem. 
Theorem 5.4. If m = n — m, then 

m 

A + (B) = {p = ^ N £ j I H e 2N o and pi > . . . > p m -\ > \p m \ } ■ 
j'=i 
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In all other cases, 

m 

I ^ ^No and fix > . . . > >o}. 

3=1 

6. The generation of the AT-spectrum 

Recall from Section [S] the involution 0(AT) = —A* on g. The Lie algebra g decom- 
poses into eigenspaces of 9 as g = { © s, where 

s = g(-l, 9) = {Ie M„,„ | 6{X) = -X and Tr(X) = 0} . 

Then, except in the case n = 2, the complexification Sc of s is an irreducible L- 
spherical representation of K. For n — 2 this representation decomposes into two 
one-dimensional representations. 
Let 

/n=SI \ 

= "n _™t Gs - 

V u n L n-mJ 

Then is A-fixed and (Ho, Ho) = 1- Define o := M.H a . The operator ad(TJo) has 
spectrum {0, 1, —1} and n = g(l, &d(H ))- 

Let Ad(fc) denote the conjugation by k. Define a map ui : Sc — > C°°(B) by 

w(30(fc) := (y, Ad(fc)i/ Q ) = /3(r, Ad(fc)ff ) - -^Tr^fci/ofc- 1 ) 

and note that 

uj(Ad(h)Y)(k) = (Ad(h)Y,Ad(k)H ) = (Y, Ad(h~ 1 k)H ) = u{Y)(h- 1 k). 

Thus a; is a AT-intertwining operator. 

Fix an orthonormal basis X%, . . . , ATdi mq of q such that X\, . . . , X,„, is an or- 
thonormal basis of b. Denote by ft = — . Xj the corresponding positive definite 
Laplace operator on B. Then 

^Uj(B) = w(/i)id, 

where 

A simple calculation then gives: 

Lemma 6.1. Let /j, = (/ii, . . . ,/i m ) € A + (B). Then 

i \ m 
, . m(n — m) \ - / 9 \ 

= 2n ^ (m' + J*i(n - 2j)J . 

3=1 

For / e C°°(6) denote by M(f) : L?(B) -)■ A 2 (#) the multiplication operator 
<? fg. Recall the notation ttq for the finite dimensional spherical representation 
of highest weight € A + (B) . 

Theorem 6.2. Let Yes. Then [n,M(u(Y))] = 2ir (Y). 

Proof. This is Theorem 2.3 in 0. □ 

For fj, e A+(B) define * M : L*(jB) ®* c "> £ 2 (#) b Y 

k) : = M(«(y))p. 

Observe that for fc 6 AT, F € Sc, and <^ € L^(B) we have 

£(fc)(M(w(F)<^) = (^(/c)o;(r))(£(fc)^) = M (w(Ad(fe)F)) 
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with Ad(k)Y £ sc and £(k)ip 6 L 2 ^(B). Hence, ^ is if-equivariant and Im^ is 
if-invariant. Define a finite subset S(fi) C A + (£>) by 

Lemma 6.3. Le£ fj, e A+(B). Then 

S(n) = {n±2e j \j = l,...,m}nA + (B). 

These representations occur with multiplicity one. 

Denote by pr CT the orthogonal projection L 2 (B) — > L 2 r (B). The first spectrum 
generating relation which follows from Theorem 16.21 see also Cor. 2.6], states: 

Lemma 6.4. Assume that fi £ A + (B). Let a € S(n), Y 6 Sc, and A 6 C. Let 
uv(Y) := pr a o M(a,(Y))| L j (B) : L»(B) -> L*(B) . (6.1) 

pr, o 7r A (y)Uj (B) = - + 2^^ AK M (F) . (6.2) 

The spectrum generating relation that we are looking for can now easily be 
deducted and we get: 



Lemma 6.5. Let /i = (/ii, . . . , pb m ) £ A + (B) and A e C. T/ien 

Vfgej 0) _ A-Mj + j - 1 _ -A + /ij - j + 1 



(6.3) 



?7/i (A) A + /ij + n — j + 1 A + fij + n - j + 1 
and T)q(X) = c(A). 

PROOF. First we apply Cm to (|6.2|) from the left, using that C m com- 
mutes with pr CT and that C m n ^ 2 oirx(Y) = K-\o0(Y)oCm = — 7T-A(X) Cm ™^ 2 - 
We then get: 

(u(cr) - wfc) + 2^^ A)r>(A - n/2)^(F) = 

- ( U (a) W ( M ) - 2=^ A)^(A - n/2K M (F) . 

As uj a s(X) is non-zero, for generic A it can be canceled out. 
Now insert the expression from Lemma |6. II to get 

W (/x + 2 CJ ) - = (^ + n/2 - (j - 1)) 

and the claim follows. The last statement follows from the fact that 7ta is irreducible 
for generic A, hence, iterated application of (|6.1| will in the end reach all if -types 
starting from the trivial if -type. □ 

Lemma RT31 tells us that the evaluation of ?7 M (A) can be done in two steps. First 
we determine the function 770(A) and then use (|6.3[) as an inductive procedure to 
determine the rest. The final result is given in the following theorem. It is presented 
in terms of T-functions associated to the cone fl of m x m positive definite matrices, 
namely, 

m 

r n (A)=rr™( m - 1 )/ 4 []r(A J -0--l)/2), A = (Ai,...,A m ) e C m . (6.4) 

3=1 
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This integral is a generalization of T m (A) in (pT5]> : cf. 16, p. 123], '7Q\ Sec. 2.2]. 
In the following the scalar parameters, which occur in the argument of Tq, are 
interpreted as vector valued, for instance, n ~ (n, . . . , n), A ~ (A, . . . , A). 

Theorem 6.6 ([57]). LetA + (B) be the sublattice in Theorem \ 5.Ji\ parametrizing 
the L-spherical representations of K, let fi — (/ii, . . . , /x m ) G A + (B), and A G C. 
Then the K -spectrum of the cosine transform is given by: 

m = ( nM/a r m (n/2) r m ((A + m)/2)) r n (fc - A)/2) 

1 ' r m (m/2) r m (-A/2) r ((A + n + M )/2) ' lD '° J 

Remark 6.7. Owing to (|3.12l) . the spectrum of the normalized cosine transform 
%? m has the simpler form 

w=(-i)'^ % r ^ ( ryf/ov ( 6 - e ) 

r n ((A + n + /i)/2) 
In the case m = I this formula coincides with (12.61). 



Remark 6.8. In Section^] we referred to the result of Vogan and Wallach on 
the meromorphic continuation of the intertwining operator J(A). This result is not 
needed for the computation of 77^ (A). Indeed, it is enough to know that J (A) is 
holomorphic on some open subset of C as that is all what is needed to determine 
rjn (A) in Theorem 16.61 We can then extend meromorphically on each if- type. 
Note, however, that this is weaker than the statement in [83] which extends C^f 
for all smooth functions. 
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